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ABSTRACT 

A complex research and development process is 
required to study instructional processes and student outcomes 
effectively. In order to study the instructional process it is 
essential to select or develop instruments that can describe a total 
event. Understanding the classroom process necessitates having a 
record of the environment, the materials, the interactions, and 
activities of the teacher and children. The first step in studying 
instructional process is to examine and specify the critical 
components of the classroom or the 'teaching program being studied • 
The next step is to identify or develop an observation instrument to 
record these critical components reliably. It is especially important 
to select appropriate statistical procedures sin^e observation data 
often form J-shaped curves that defy analysis using conventional 
parametric procedures. (MV) 
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THE IMPORTANCE OF MULTIPLE DATA COLLECTION INSTRUMENTS '.-THEN 
DESCRIBING THE EDUCATIONAL PROCESS 

The research that I find most compeLLing is Che search for relation- 
ships between instructional processes and growth in the person receiving 
the instruction. Several studies have been conducted in the 1970s iden- 
tifying instructional processes that can be validly related to outcomes 
* for the learner. Most notable of these are Brophy and Evertson (1974), 

McDonald and Ellis (1976), Soar (1973), and Stallings and Kaskowitz (1974). 
When the findings of these studie.<= ^re carefully analyzed, the results 
are seldom found to be contradictory, even though researchers have used 
di\fferent samples and different instruments. 

It is now becoming possible to apply the findings of these researchers 
to teacher training. Such an effort is now being carried out at Stanford 
University under the direction of Nathan Gage. The instructional tech- 
niques being developed are expected to produce specific student outcomes. 
However, this paper will be liruited to the discussion of the complex re- 
search and development process that is required to study instructional 
processes and student outcomes. 

To learn the most from a study of process, it is essential to select 
or develop instruments that can describe a total event. A classroom can 
be likened to a stage on which a play is taking place. The physical 
structure of the classroom is the setting for ttie play; the materials and 
equipment are tb» props. The scenes of the play, tnen, are the various 
activities and grouping of characters, and the plot consists of sequences 
of interaction between the teacher and the children and among the children 
themselves. All of the components are necessary to an understanding of 
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Che play. Similar Ly, to comprehend the classroom process, it is neces- 
sary CO have a record of Che environmenc, Che macerials, che accivicies 
of Che ceacher and children, and che inceraccions chac occur as chey 
parcicipace in the accivicies. 

A firsc seep in scudying inscruccional process, Chen, is Co examine 
and specify che cricical components of che classroom or che ceaching 
program being scudied: whac is che environmenc, whac are che accivicies 
and groupings, and whac inceraccions are caking place? The nexc seep is 
Co idencify or develop an observacion inscrumenc Co record che idencified 
critical componencs reliably. For SRI ' s recenc scudy of Follow Through 
(Scallings and Kaskowitz, 1974), we developed an observation instrument 
to record the critical components of seven Follow Through sponsors. We 
did this by visiting a model classroom of each sponsor in a specified 
city to obtain descriptions of the physical environment, the activities 
that occurred, the classroom management strategies, and the interactions 
between teachers and students. From the ethnographic recordings, we 
developed a coding system. The variables we developed from the coding 
system were sent to each sponsor to check whether the critical components 
of their model could be. recorded in this system. Because one model had 
a traveling teacher (one who moved from child to child assisting and pro- 
sliding feedback), we had to develop a code to show when movement around 
the classroom occurred. Because another sponsor directed teachers to 
g-ive tokens for correct responses to questions about the subject matter, 
we had to provide a code to indicate when a token was given along with 
praise for the correct response . Fortunately for the project, we were 
able to collect data and refine our observ"?tion system over four data 
collections in four years. Each year we were able to identify those 
variables that distinguish sponsors from each other, to identify codes 
that were unreliable and redefine or delete them, to improve the training 
procedures, to select a large enough sample to assure a wide range of 
process and outcome scores, to develop an efficient data processing sys- 
tem, and to select appropriate statistical procedures. 

This last point is very important because observation data can not 
be treated I^ke test data. Observation data often form. J-shaped curves 



chac defy analysis using conventional parametric procedures. For the 
Follow Through data, we used nonparame trie procedures to examine program 
implementation of first ana third grade classrooms in five different 
locations for each sponsor. These uonparametric procedures using quintiles 
are reported in Stallings SRCD Monograph (1975). The validity of the 
statistical procedures for examining implementation was borne out in the 
following way. The Behavioral Analysis Model at: University of Kansas had 
the first and third grade classrooms receiving very similar implementation 
scores on their set of variables in four out of five of their model cities. 
Director of the prcgron, Don Bushell, was asked about the teachers or 
training in th-. variant city. The question- was : were the observers 
reporting inaccurate d^':^ or were the teachers in that city truly different 
from the teachers in the other four cities? Bushell' s response was that 
the particular teachers in our sample for that city had not reached the 
required proficiency and were therefore not yet credentialed for his 
program. The teachers in the other four cities had achieved proficiency 
and were credentialed by Bushe'l. Thus, one could conclude the observa- 
tion system was reporting accurate information about the Bushell program 
zw: that the stati.Jtical procedures being used were appropriate for mea- 
suring inplementatiou. Two other f'poasors also reported agreement on 
th.;ir ir^.sr uments with th3 SRI data for s:jecific classrooms which were 
not well implemented. 

Other information about implementation comes from the teachers them- 
selves. We had teachers rate themselves .on a 44-point scale of structure 
and flexibility, and we found the teachers* self -ratings similar to our 
observations of structure. For example, teachers who were observed to 
emphasize subject matter also r.^ted themselves high on this item. A more 
substantial method of examining teacher report of implementa*:ion is that 
developed by Gene Hall (1976). In Hall's work, the six levels of imple- 
mentation are defined through interviews. These levels can then be cor- 
related with student growth. Gene Hall and Susan Loucks are currently 
conducting a study of a mathematics curriculum in Denver, Colorado, and 
they are using the Hall teacher interview and the Stallings/SRI observa- 
tion system to assess levels of implementation. It will be of great 
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interest to see how teacher self-report data confirm or contradict obser- 
vation dar ^ '^i. hew fully the curriculum is being implemented. 

Outcomes 

Once the program to be assessed has been defined and the instruments 
have been developed or selected, the next step is to examine what effect 
the instructional processes have on the learner. Historically, the pri- 
mary outcomes for learners have been standardized achievement test scores. 
However, during the 1960s and 1970s, many educational programs were aimed 
at educating the whole child. To determine whether specific programs are 
achieving their goals, social and emotional ou .comes must be measured 
as well as cognitive outcomes. Unfortunately, measurement tools have not 
developed at the same rate as the educational programs. Schools have 
programs dedicated to developing curiosity, self-esteem, social awareness, 
group cooperation, problem-solving ability, the ability to plan, execute 
the plan, and evaluate the work, but researchers do not yet have satis- 
factory methods for evaluating the effectiveness of such programs. 

In the Follow Through 1974 study, in addition to the Metropolitan 
Achievement Test, the third grade children were given the Raven's Pro- 
gressive Matrices to assess nonverbal problem-solving ability, the In- 
tellectual Achievement Responsibility Scale, and the Coopersmith Self- 
Esteem Inventory to assess how the, child felt about himself. Absence 
rate was used as an indicator of attitude toward school, based on the 
assumption that children who like school are absent less often. To 
assess such variables as task persistence, question asking, and group 
cooperation, a random sample of children was observed. Since it is dif- 
ficult to define such attributes as self-esteem or attitude toward school, 
it is important whenever possible to have several measures (observations, 
in addition 'to tests) of these interesting but elusive variables. 

Corre la tion 

In order to estimate the effect of the Follow Through sponsor's 
model, x^e used the score the children obtained on the Wide Range 
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Achievemenc Tesc when they entered school in kindergar ten. Holding this 
score constant allowed us to examine the effect of the model over two 
years for the first grade and over four years for the third grades. These 
few tests left much to be desired in measuring important expected child 
outcomes in the Follow Through classrooms. However, test development is 
moving ahead and there are more tests available now than there were in 
1973, when the Follow Tnrough data were collected. 

Since so little was knox>ni about how instructional processes are 
related to learner outcomes, we took the shotgun approach in the Follow 
Through observation evaluation and ran partial correlations using 340 
process variables x^ith each outcome measure. These analyses were con- 
ducted using classroom means for 108 first grades and 58 third grades. 
These classrooms exhibited a x^tde range of instructional processes and 
student outcome scores. Out of the 340 possible correlation, the numbers 
significant at the 0.05 level were 118 in reading, 114 in math, 58 in 
problem solving, and 65 in absence rate. The correlations were signifi- 
cant in both a positive and negative direction; that is, there were as 
many suggestions about what a teacher ought not to do as there were about 
what a teacher ought to do. 

With so many significant findings, we attempted to refine these 
through a stepwise regression that could identify the smaller set of 
variables that accounted for most of the variance. This procedure worked 
well in that tht: entering school ability of the children was accounted 
for first, allowing us to examine the variance in test scores explained 
by the process variables. We found that as much (or more) of the test 
score variance was explained by the instructional processes as by entering 
ability. Those processes contributing the most to the explanation included 
variables describing grouping, organization, curriculum, and instructional 
interactions. Because tha total classroom environment was obser\'ed and 
recorded, a more complete pictur^ of what is important co classroom 
insLruction could emerge. 

For example, I think othe:: studies of reading, which have not found 
a significant relationship, have addressed single dimensions such as 



curriculum, or the 45-minuce formal reading period, or grouping patterns, 
or individualization. Analyses of single program components are too limited 
in ':heir view. Instructional learning occurs in the context of a living en- 
vironment. It is not just the materials that affect the learners; it is h^w 
Che materials are used. It is not just the length of time spent in the 
formal reading period, but how much actual time each child spends in the 
act of reading all day, Tt is not just how teachers group children, but 
how teachers manage the groups to keep all children working on their 
tasks for the maximum amount of time, 

A current research study at SRI is allowing us to study basic reading 
skills taught to junior high and high school students. This. time we may 
not need to use a 12-gauge shotgun. We know a little more of what to 
look for. We can form hypotheses based upon prior research. We shall 
examine time spent cn task, length of period, number of teachers and aides, 
class size, classroom management, individualized curriculum, instructional 
patterns, classroom control, positive or negative affect. All of these 
are variables identified as being related to child learning in the re- 
search of Brophy and Evertson (1974); McDonald (1976), Wiley (1975), and 
Stallings and Kaskowitz (1974). We shall still take some exploratory 
shots in our hunting expedition, because at the upper school level we 
expect that self-esteem^ peer pressure, and absenteeism is likely to have 
a greater effect on the learning- to-read process. We are eager to explore 
this unknown area and find what instructional processes are related to 
achievement, self-esteem, and absenteeism at this level. To do this we 
shall compute partial correlations ^ using the instructions process vari- 
ables and the outcomes of interest. Any piece of the puzzle we may iden- 
tify that contributes to the picture of ef.^-'ective teaching and learning 
can form the basis for hypotheses for more carefully controlled experiments 
where teachers are taught to use the processes identified as beneficial 
to student learning. We will use several instruments to examine our 
hypotheses about the classroom process at the junior high school level. 
These include the Comprehensive Test of Basic Skills (CTBS), the FIRO-B, 
the Student Classroom Environment Scale, the Student Rating of Teachers, 
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a Teacher Quest ionnaire, and a newly developed observation of interactions 
and classrooin activities. These will allow us to study process and out- 
come from the point of view of the student, the teacher, and the observer. 

In a sense this research is at. the dawn--a few rays of light are 
coming across the horizon. We, the research community, have learned in 
Che past that ttacher characteristics such as age, education, experiance, 
or SES, do not explain whv some teachers are successful and others are 
not. We have learned that curricula alone do not explain why some children 
learn and others do not. We have learned that school level variables do 
not explain test score differences. We have learned that what teachers 
do in the physical setting of their classrooms--their instructional t ch- 
aique--does make a difference in what children learn. Our ability to 
measure instrucnional process and a variety of student outcomes is pro- 
gressing. We can use several measurements--observation, interview, and 
tests--to check our findings. Since 1970, the way teachers and students 
are evaluated has been changing. To go forward, it is important for 
researchers in this field to share their findings. Whenever possible 
we should use common metrics and variables so that we can build a body 
of knowledge regarding instructional process and student growth. Research 
in this area has just begun, but it is a new day and we are all fortunate 
to be a part of this challenging work. 
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